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ABSTRACT 



This paper reviews the literature on academic quality 
programs and examines issues faced at one institution in which two 
initiatives emphasizing student outcomes and faculty performance as 
indicators of quality are underway. In its review of the literature the paper 
focuses on concepts such as total quality management, peer or student 
evaluation of faculty, organizational change, and human and technological 
factors; and it defines the characteristics of quality academic programs as 
including a culture of shared purpose, an active learning environment, high 
expectations for students and faculty, continuous feedback for both students 
and faculty, adequate resources, and good administrative support. At the St. 
John's University (New York) Centers of Excellence, resource allocation is 
integrated with assessment of student outcomes and faculty performance. The 
university uses four categories of measurement to select academic departments 
as centers of excellence: input (ability to attract high-quality students, 
excellent facilities, favorable faculty/student ratio) ; process (programs 
demonstrating long-term interest and need, and outstanding curriculum) ; 
value-added features (evidence of student satisfaction) ; and output 
(productive faculty, graduate placement, and evidence of student satisfaction 
after graduation) . Key to the success of the program is careful attention to 
the balance between the technical (data analysis) , organizational (shared 
values), and personal frames. (Contains 30 references.) (CH) 
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ABSTRACT 

Strategic Planning, Assessment and Accountability: 

Their Impact on the Establishment of Centers of Excellence in Academic Departments 

The ongoing demands for universities to be accountable for student achievement and 
faculty performance has led to the growth of the outcomes assessment movement. Strategic 
plans are increasingly incorporating measurement into their texts, leading to debates as to what 
to measure, how to proceed and how to interpret the results. This discussion will review the 
literature on quality academic programs and examine the issues faced at one institution, where 
two initiatives are underway that emphasize student outcomes and faculty performance as 
indicators of quality: academic program review and the establishment of "Centers of 
Excellence" for outstanding programs. Support for these and other initiatives is provided by 
the Office of Institutional Research, particularly in the work of key committees and working 
groups. The need for assessment to feature multiple methodologies, process management and 
faculty ownership is explored. 
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The challenge of accountability facing higher education at the system and institution 
levels has resulted in a climate of assessment that has the potential to influence the way that 
faculty perceive their roles, particularly in relation to their students. Tension abounds between 
the dual pressures to be successful at an active research enterprise while reaching out to 
students, especially undergraduates. Faculty are certainly not strangers to the need to juggle 
teaching and research responsibilities, but many are only now realizing that public pressures— 
formal and informal— on administrators to account for their classroom time, publication 
productivity, and student performance are intruding upon matters that have traditionally been 
left to their personal professional judgment. 

This paper will review the literature on planning and outcomes assessment with a focus 
on characteristics of quality academic programs. A discussion will follow of the criteria for 
designated “Centers of Excellence” at St. John’s University, New York, and the role of the 
Office of Institutional Research in coordinating initiatives that support the various processes of 
assessment. St. John’s is accredited by the Commission on Higher Education/Middle States 
Association (hereafter, Middle States); therefore, references to the criteria of that regional 
accrediting agency are made throughout the discussion. 

Literature and Conceptual Framework 

Middaugh (1995) credits three developments as having combined to make faculty 
productivity a key concern for American higher education: rising costs, accountability, and 
restructuring. At the core of these concerns is the pressure to improve student performance by 
focusing attention on the learner. Even this emphasis on the student takes on varying foci, 
including such concepts as Total Quality Management (TQM) as a function of the faculty role 
(Chaffee & Sherr, 1992) and faculty inputs as responsible for learning productivity (Johnstone, 
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1993). Another prominent concept is that of faculty evaluation by peers and/or students (Elton 
& Carey, 1989; Gullatt & Weaver, 1995; Tucker, 1991). The focus of the faculty member's 
professional orientation is the academic discipline in which he/she conducts teaching and 
research activities; as the member of the campus community, this focus is centered in the 
academic department. Efforts to achieve and promote academic excellence must of necessity 
have faculty ownership, which may result in tension between the legitimate dual needs for 
standardization (institution) and specialization (department). It is essential for institutional 
researchers and other administrators to keep in mind that the process deserves equal 
consideration between inputs and outcome measures in order to improve quality (Tucker, 

1991; Nordvall & Braxton, 1996). 

The reality of life in all organizations (including society) at this pivotal point in the 
Twentieth Century is that change— and with it, restructuring— will play a constant role in the 
experience of the participants (Ramaley, 1995). A key to understanding organizations 
undergoing transformation is to acknowledge that there are multiple dimensions to be 
considered when seeking to capture faculty performance and student outcomes in a program or 
department, particularly when their relationship is the focus of inquiry. The three-part 
framework (i.e.. Technical, Organizational, Personal) proposed by Linstone and Mitrolf 
(1994) offers a solution by proposing that each frame has limitations but together they provide 
a balanced look at organizations necessary for change. 

The Technical frame seeks to solve problems through data analysis and model-building 
while the Organizational frame examines the nature of culture through the lens of process and 
values created by participants. The Personal frame provides a filter through which participants 
perceive the reality of organizational events. Linstone and Mitroff emphasize the linkages of 
each combination of pairwise frames and stress the inseparability and interdependence of the 
triad components. It is an organizational concept similar to the notion of triangulation of 
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qualitative and quantitative data analysis long advocated by research methodologists (Denzin, 
1970; Chronbach, 1975; Guba & Lincoln, 1981). 

Acknowledgment of the differing foci and contributions of the three perspectives— 
individually and in combination— is a key step toward an integrated approach to decision- 
making and evaluation that is applicable to higher education. The case that provides the basis 
for discussion in the Linstone and Mitroff book (i.e., the Exxon Valdez oil spill) was 
influenced by a combination of human and technological factors. It illustrates well the 
complexity of both the challenge of decision-making and of the competing perspectives. 

As administrators and faculty grapple with the formidable task of making their 
institutions accountable to students, parents, government, press and public, they need to keep 
in perspective that what may appear to be a quick fix— providing data on student performance 
on standardized tests— does not portray either the achievement of students or the efforts of 
faculty. To paraphrase Linstone and Mitroff’s suggestion that to combine the technical, 
organizational and personal perspectives is to change in focus from an “error-inducing to a 
safety-reinforcing system” (p. 78), higher education needs to change its focus from a 
deficiency-reporting to an achievement-sharing system. 

The assessment movement that has taken higher education by storm in the last decade 
has seen the infusion of business concepts such as TQM into its vocabulary. Such concepts 
cannot be imposed by administrators but need faculty involvement at the classroom and 
curriculum level. According to Chaffee and Slierr (1992, p. 93), quality cannot be perceived 
by suspicious faculty as “another management fad from the evil empires of business” or an 
administrative cost-cutting strategy; administrators need “to take seriously the need for a long- 
term, change-oriented commitment to TQM for themselves.” 
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Planning and Outcomes Assessment 

Modern forms of planning in higher education emanated from the annual budget 
processes of the 1950s. As institutions grew in size and complexity, competition intensified, 
and the environment became more uncertain, emphasis began to shift to long range plans for 
institutions (Green & Jonas, 1997). Throughout the 1980s and into the 1990s, demands by 
governmental agencies, accrediting agencies, media, parents, students, and the general public 
for accountability and increased productivity have forced educational institutions to examine 
ways to strategically plan, assess, and improve the quality of their educational programs 
(Heaney, 1990). 

Elements common in definitions of strategic planning include: a continuous and 
systematic process of making decisions about intended future outcomes; organizing the efforts 
needed to implement decisions; and measuring and evaluating the results of the decisions 
against expectations through organized, systematic feedback (Drucker, 1980). Outcomes 
assessment is clearly an integral part of any effective strategic planning process. 

The primary purpose of outcomes assessment is to measure institutional effectiveness 
and student outcomes, and to use the results for institutional improvement. Since 1985, most 
higher education institutions have been required by state mandate or regional accreditation 
requirements to put assessment programs in place (Borden & Banta, 1994). For example. 
Middle States expects each institution to develop guidelines and procedures to assess student 
learning outcomes and overall effectiveness. Outcomes assessment is defined by Middle States 
(1994, p. 16) as involving 

gathering and evaluating both quantitative and qualitative data which demonstrate 
congruence between the institution’s mission, goals, and objectives and the actual 
outcomes of its educational programs and activities. The ultimate goal of outcomes 
assessment is the improvement of teaching and learning. 
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At many institutions, resource allocation has remained largely an administrative issue. 
On the other hand, assessments of student achievements have drawn faculty directly into the 
process. These efforts generally continue along separate paths. The issue is further 
complicated at institutions that add faculty performance to student outcomes as indicators of 
quality. As the Middle States standards indicate, “colleges and universities that include 
research, scholarship, and service as major institutional purposes should assess effectiveness in 
these areas as thoroughly as they evaluate learning outcomes " (1994, p. 17). 

A recently released Carnegie report, Scholarship assessed: Evaluation of the 
professiorate (Glassick, Huber & Maeroff, 1997) suggests that not only is assessment of 
faculty critical, but that contrary to the conventional wisdom in academe, common standards 
can be used to assess the three components of faculty work. The six standards cited in the 
report include: clear goals, adequate preparation, appropriate methods, significant results, 
effective presentation, and reflective critique (Magner, 1997). Results of a related Carnegie 
survey of four-year colleges showed that faculty evaluation policies are in place in most 
institutions. Almost all institutions surveyed are using the following policies to evaluate 
teaching: systematic student evaluation of classroom teaching; self-evaluation; and peer review 
of teaching materials and classroom performance. Interestingly, evidences of student 
achievement are used by less than one-fourth of institutions. In terms of research and service, 
self-evaluation is the method used by most institutions. 

Characteristics of Quality Academic Programs 
Culture of Shared Purpose 

One of the most important attributes of quality programs is a shared sense of purpose 
among students and faculty. This concept has been described in many ways and in various 
contexts, but the essential ingredient is harmony with clearly articulated institutional and 
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departmental mission and goals. Banta (1993) calls for the need for continuous communication 
and the right fit between student and institution. Frequent personal exchanges between faculty 
and students are facilitated by shared interests and values (Pascarella & Terenzini, 1991). 

As one of the Seven Principles for Good Practice in Undergraduate Education , a report 
resulting from a study sponsored by the Johnson Foundation and the Lilly Endowment, 
student-faculty contact should not be limited to the classroom but continue after class as well 
(Chickering, Gamson, & Barsi, 1989), a point echoed in a 1995 report from the Education 
Commission of the States (ECS). In their study of master’s education, Conrad, Haworth and 
Millar (1993) extend unity of purpose to alumni and employers in addition to students, faculty, 
and administrators. Conditions should be fostered that facilitate dialogue where leaders actively 
listen to all constituencies. Students, alumni and employers have a place on committees and 
advisory boards. 

Active Learning Environment 

The human dimension extends to the total learning environment in order to make the 

acquisition of knowledge relevant for students and faculty alike. Students need frequent 

feedback both within class and during the advisement process. When faculty take the additional 

time necessary to plan and guide discussions, field work, internship and individualized 

learning experiences, students synthesize and apply their new knowledge (ECS, 1995). 

According to Pascarella and Terenzini (1991, p. 648), faculty and administrators should 

shape the educational and interpersonal experiences and settings of their campus in 
ways that will promote learning and achievement of the institution’s educational goals 
and to introduce students to become involved in those activities, to exploit those 
settings and opportunities to their fullest. 

Faculty can set the tone for such an environment by the example of participation in 
their respective professional/disciplinary associations and being active scholarly publishers. 
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Faculty actively publishing and participating in association activities who recruit students to 
become involved in these processes enhance the quality of their students’ education by 
instilling a passion for knowledge at the cutting edge, which translates into appreciation for 
life-long learning and a sense of perspective as to where learning fits into the scheme of life 
(Conrad, et al., 1993). 

High Expectations for Students and Faculty 

Quality education at all levels requires that students be given goals that are placed at 
“high but attainable levels” (ECS, 1995, p. 17); it is essential that this opportunity be extended 
to everyone, including “the poorly prepared, . . . those unwilling to exert themselves, and . . . 
the bright and well motivated” (Chickering, et al., p. 1 1). Expecting students to reach beyond 
the relatively safe confines of amassing facts involves a degree of risk-taking on their part, 
with which some may not be comfortable. While to take risks is to achieve greater success in 
the end, students need confidence to realize that what appears to be short-term failure is an 
important part of the long-term learning process. Therefore, it is essential for faculty to create 
an environment that encourages risk but in which students feel secure (Conrad, et al., 1993; 
ECS, 1995). 

Faculty involved in quality programs need support for the labor-intensive efforts 
required to develop and maintain such experiences for students. The demands placed upon 
faculty to intensify the time and energy devoted to developing activities incorporating active 
learning and individualization while extending a publication record is complicated by the 
necessity to continuously upgrade knowledge of teaching technologies and the influx of 
students at varying levels of ability with differing learning styles. Banta (1993) notes that 
successful implementation of such programs requires institutional support of the faculty, 
specifically in the areas of faculty development for improvement of teaching and evaluation. 
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Continuous feedback for students and faculty alike is an integral part of a quality 

academic program. Students bring different talents and backgrounds to their undergraduate and 

graduate education, and these need to be enhanced and supplemented by new ways of 

discovery. Chickering, et al., (1989, p. 1 1) notes that 

brilliant students in the seminar room may be all thumbs in the lab or art studio. 
Students rich in hands-on experience may not do so well with theory. Students need the 
opportunity to show their talents and learn in ways that work for them. Then they can 
be pushed to learning in new ways that do not come so easily. 

Early and frequent assessment permits faculty to determine learning styles among the class and 
to make necessary adjustments to account for this diversity (ECS, 1995). 

Banta (1993) includes feedback for faculty in her recommendation that instruction and 
evaluation be included in methods of improvement of teaching and learning. Many institutions 
implement some form of student evaluations of faculty and nearly all require peers, chairs 
and/or deans to provide assessment as part of the reappointment, promotion and tenure 
process. It is important that such measures include items as sensitivity to varying student needs 
and abilities, appropriateness of pedagogical techniques, and accessibility/approachability of 
the faculty member to students. 

Resources and Administrative Support 

The best-intentioned and hard-working faculty and administrators cannot develop or 
enhance quality academic programs without support of several types. The most obvious form 
ot assistance (i.e., funding) is needed to meet a variety of needs, including (but not limited to) 
initiating new programs, student financial aid, facilities construction and enhancement, and 
professional development of faculty, administrators and staff (Chickering, et al., 1989). 

Beyond the obvious need tor funding (and in many ways, equally critical) is an 




12 



appropriate institutional infrastructure. This organizational foundation should be grounded in 
bona fide recognition of disciplinary differences intertwined with the concomitant varieties of 
scholarly output and service that result from faculty dedicating time and effort to participate in 
quality programs. Such recognition should be extended to the faculty reward structure, 
particularly as it is operationalized by promotion and tenure committees (Conrad, et al., 1993). 

In addition to the presence of the above components being necessary for support, it is 
important that certain impeding factors be absent. Administrative attention must be dedicated 
to the elimination of barriers to quality work on the part of faculty, administrators and staff. A 
climate of cooperation to fix the process rather than blaming those who work within it is 
necessary to remove obstacles (Chaffee & Sherr, 1992). The policies and procedures in place 
for both faculty and students must minimize bureaucratic regulations in concert with the 
mission and goals of the institution, academic unit and the program in question (Chickering, et 
al., 1989). Results of the study by the Education Commission of the States (1995) indicates 
that students consider “understandable and efficient administrative processes” (p. 13) to be 
particularly desirable. 

Centers of Excellence 

At our institution (St. John’s University, New York), resource allocation is being 
integrated with the process of student outcomes and faculty performance through the 
establishment of Centers of Excellence. The first five Centers to be designated are Biological 
Sciences; Library and Information Science; Psychology; Speech-Language Pathology and 
Audiology; and Educational Administration. As outlined in the Strategic Plan (1995, p. 25), 
“the University will seek new administrative initiatives to support and strengthen an . 
institutional culture of academic excellence, based on our mission and values, and 
characterized by scholarship, research, teaching excellence, openness, innovation, flexibility, 
vibrancy, and collegiality.” 
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Higher education institutions use some or all of four categories of measurements to 
define quality: input; process; value-added; and output (Bergquist, 1995). The criteria for 
selection of an academic department as a Center of Excellence at our institution can be 
grouped into these four categories. 

Inputs include purposes and objectives reflective of the University’s mission and goals, 
ability to attract high quality students, excellent facilities to meet the needs of the program, 
and an academically favorable faculty/student ratio. Processes include programs for which 
evidence exists of long-term interest and need on the part of the public, and a cutting edge 
curriculum as attested by external reviewers. 

Value-added features include evidence of student satisfaction within the program. 
Finally, outputs include a core of highly productive faculty (both in terms of teaching and 
scholarly activity), placement of graduates in positions of leadership and/or graduate schools of 
distinction, and evidence of student satisfaction after graduation. 

In addition, the Strategic Plan identifies the primary strategic goal as fostering a 
culture of academic excellence in which “continuous quality improvement must remain a 
priority (p. 25). This will not be easily accomplished; continuous process improvement (CPI) 
and other similar methods of appraising performance are generally considered by faculty to be 
more suited to administrative than to academic areas (Borden & Banta, 1994). In the past, 
academic processes have tended to inhibit change. Traditionally, higher education institutions 
have been evaluated on inputs (i.e., level of resources, quality of the freshman class) to a 
much greater extent than on any other criteria (Astin, 1983). 

A radically different approach will be needed to measure institutional and departmental 
effectiveness through the lens of quality improvement. The primary motivation for this 
approach is to ensure that the institution achieves its vision and mission. As a part of this 
process, faculty and other stakeholders must be involved in the planning process and share in 
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the power, information, and rewards (Freed, Klugman, & Fife, 1997). In particular, faculty 13 
must be convinced that assessment of student outcomes and faculty performance will facilitate 
continuous process improvement efforts and can increase students' learning. Faculty must be 
invited to take a leadership role in developing performance indicators and other measures of 
assessment. Otherwise, they might perceive such initiatives as an infringement on their 
autonomy (Nedwek & Neal, 1994) and are unlikely to implement any resulting 
recommendations (Schmidtlein, 1990). 

Implications for Institutional Research 

One of the key principles on quality improvement identified by Freed, et al. (1997) is 
the need for the systematic collection and reporting of data. The institutional researcher has a 
pivotal role as a facilitator in this process. This includes collection and analysis of data to assist 
with assessment of needs and expectations, as well as assessment of current levels of 
performance and any gaps. The institutional researcher will also assist in the development and 
implementation of systems to measure and monitor outcomes (Green & Jonas, 1997) and in the 

ongoing analysis of appropriate information in a timely and reliable manner (Borden & Banta, 
1994). 

At St. John’s, this process is already underway. As a part of the university-wide 
outcomes assessment effort, a Learning Outcomes Task Force has been established. 

Comprised of faculty from all academic units, the task force has developed special expertise in 
various methods of assessing outcomes. Members serve as consultants to their colleagues, 
helping them to develop measurable goals for all academic programs and appropriate ways of 
measuring achievement of these goals. These efforts are being coordinated with ongoing 
academic program reviews being conducted under the direction of the Office of the Provost, 
monitored by the academic deans and carried out by the program faculty, who have ownership 
of the process. The institutional researcher serves as a resource person on this committee and 
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assists the faculty with collection and analysis of assessment data. By ensuring that 
information is presented in a consistent manner for all departments, the institutional researcher 
increases the likelihood of acceptance and use by faculty members. This same principle will be 
applied to the provision of data to the initial and prospective Centers of Excellence. 

Two other related initiatives are underway, the first dealing with the administration of 
surveys. Like other institutions, the university’s operational information system has very 
limited data to measure student outcomes or faculty performance (Schmidtlein, 1990). Many 
departments and divisions have recognized that surveys serve as a valuable way of collecting 
outcomes data. Student satisfaction surveys, follow-up studies, surveys of recent graduates, 
and others are being used to try to determine the effect of the university experience on the 
lives of the students. 

However, as surveys have proliferated, problems of reliability, validity, and 
redundancy have arisen. A Survey Committee, comprised of faculty members with expertise 
in survey design and analysis, and chaired by the Director of Institutional Research, has been 
created. This committee has developed a survey checklist, and is responsible for review and 
approval of all proposed surveys. Committee members are available to assist faculty members 
and departments in all aspects of the survey administration process. The committee also 
proposes to continuously involve the wider academic community in this process by 
communicating plans for future institutional surveys (e.g.. Student Satisfaction Survey), 
requesting input in the process, and providing report summaries on a timely basis. These 
efforts should help to ensure improved faculty participation in questionnaire distribution when 
appropriate, acceptance of survey results, and willingness to implement the recommendations. 

The Office of Institutional Research is also spearheading the second initiative, a Data 
Integrity Committee. Comprised of a cross-section of administrative, academic, and technical 
representatives, this committee provides a forum for understanding different data needs, and 




16 



sharing and resolving concerns abonl the integrity of institutional data. This process helps to 

ensure the accuracy and consistency of institutional data that forms the basis of retention, 
graduation, and other outcomes analyses. 



Conclusion 

The Centers of Excellence will play a central role in the university's efforts to become 
a “model for distinctiveness in the 21« century” (Strategic Plan , p. 8). Freed, et al. ( 1997 ) 
suggests that continuous process improvement implies that employers and other stakeholders 
should be included in the assessment process in order that educational programs be 
continuously redesigned based on a combination of faculty knowledge and expertise with 
employers' expectations. Students' evaluation of teaching effectiveness will be introduced 



university-wide in fall 1998. The use of these and other assessment tools by faculty in the 
Centers of Excellence can serve as a model for the rest of the university. 

The Office of Institutional Research has assumed a leadership role in efforts to achieve 
the University's goal of fostering a culture of academic excellence through continuous quality 
improvement. It is through toe efforts of toe Director working with faculty, toe Provost, the 



deans and other academic administrators that toe goals outlined in toe .Sbatevic Pl,„ « ^ 
addressed. Key to toe spirit of shared purpose and cooperation has been the concerted effort to 
keep in perspective toe mix of the technical, organizational, and personal frames outlined by 
Lins, one and Mitroff ,1994). „ is only through careful attention the balance between the 
technical (data analysis) and organizational (shared values derived through a participatory 

process) frames tha, toe critical personal perspective (i.e., toe sense of individual stakeholders 
of the reality of organizational events) be maintained. 
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